In response to this:

What this PR does / why we need it:

When the etcd recovery job fails but etcd self-heals, the EtcdRecoveryJobFailed condition was never cleared. This caused the OpenShift Console to display a stale error message ("Error in Etcd Recovery job: the Etcd cluster requires manual intervention.") on the HostedCluster overview page, even when the cluster was fully healthy (Available=True, Degraded=False, EtcdAvailable=True).

This fix adds two checks in reconcileETCDMemberRecovery:

• When a failed recovery job exists but the etcd StatefulSet is fully available (3/3 replicas), clean up the job and clear the condition.
• When no failing etcd pods exist and etcd is healthy, clear any stale EtcdRecoveryJobFailed condition.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-84577

Special notes for your reviewer:

• The etcd recovery feature is gated behind ENABLE_ETCD_RECOVERY env var and only applies to managed, highly-available etcd clusters.
• Both fix paths were verified on a live KubeVirt HCP cluster by simulating the stale condition and confirming it gets cleared.
• The etcd_manual_intervention_required metric (which reads this condition) will also correctly reset.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Walkthrough

When an existing etcd recovery Job finishes unsuccessfully, the reconciler now fetches the etcd StatefulSet; if the StatefulSet reports ReadyReplicas==3 and AvailableReplicas==3 it deletes the recovery Job and related objects, clears the EtcdRecoveryActive condition (Status=False, Reason=AsExpectedReason) on the HostedCluster, updates HostedCluster status, and returns without marking manual-intervention. If the StatefulSet is not fully ready, the reconciler retains the prior failure handling (EtcdRecoveryJobFailedReason). Separately, when no failing etcd pod is detected and the StatefulSet is fully available, the reconciler clears a stale EtcdRecoveryActive failure condition only if its prior Reason was EtcdRecoveryJobFailedReason, updating status. A unit test verifies these behaviors.

Sequence Diagram

sequenceDiagram
    participant Reconciler
    participant ETCDJob as ETCD Recovery Job
    participant ETCDSet as ETCD StatefulSet
    participant HostedCluster as HostedCluster Status

    Reconciler->>ETCDJob: Check if Job exists and failed
    alt Job Failed
        Reconciler->>ETCDSet: Fetch StatefulSet readiness
        alt ReadyReplicas==3 && AvailableReplicas==3
            Reconciler->>ETCDJob: Delete recovery Job and objects
            Reconciler->>HostedCluster: Set EtcdRecoveryActive=False (Reason: AsExpectedReason)
            Reconciler->>HostedCluster: Update status
        else StatefulSet Not Fully Ready
            Reconciler->>HostedCluster: Set EtcdRecoveryActive=True/Failed (Reason: EtcdRecoveryJobFailedReason)
            Reconciler->>HostedCluster: Update status
        end
    else No Failing Pod Detected
        Reconciler->>ETCDSet: Check if fully available
        alt StatefulSet Fully Available
            Reconciler->>HostedCluster: If prior Reason==EtcdRecoveryJobFailedReason, clear EtcdRecoveryActive (Status=False, Reason=AsExpectedReason)
            Reconciler->>HostedCluster: Update status
        end
    end

❌ Failed checks (2 warnings)

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

• X
• Mastodon
• Reddit
• LinkedIn

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@@            Coverage Diff             @@
##             main    #8406      +/-   ##
==========================================
+ Coverage   39.85%   39.88%   +0.02%     
==========================================
  Files         751      751              
  Lines       92554    92582      +28     
==========================================
+ Hits        36889    36922      +33     
+ Misses      52992    52976      -16     
- Partials     2673     2684      +11     

... and 1 file with indirect coverage changes

Flags with carried forward coverage won't be shown. Click here to find out more.

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

In response to this:

What this PR does / why we need it:

When the etcd recovery job fails but etcd self-heals, the EtcdRecoveryJobFailed condition was never cleared. This caused the OpenShift Console to display a stale error message ("Error in Etcd Recovery job: the Etcd cluster requires manual intervention.") on the HostedCluster overview page, even when the cluster was fully healthy (Available=True, Degraded=False, EtcdAvailable=True).

This fix adds two checks in reconcileETCDMemberRecovery:

• When a failed recovery job exists but the etcd StatefulSet is fully available (3/3 replicas), clean up the job and clear the condition.
• When no failing etcd pods exist and etcd is healthy, clear any stale EtcdRecoveryJobFailed condition.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-84577

Special notes for your reviewer:

• The etcd recovery feature is gated behind ENABLE_ETCD_RECOVERY env var and only applies to managed, highly-available etcd clusters.
• Both fix paths were verified on a live KubeVirt HCP cluster by simulating the stale condition and confirming it gets cleared.
• The etcd_manual_intervention_required metric (which reads this condition) will also correctly reset.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Bug Fixes

• Enhanced error messages during in-place upgrades to include degraded node names and specific failure reasons.

• Improved etcd recovery handling to automatically clean up recovery resources when etcd returns to healthy state, reducing manual intervention needs.

• Tests

• Added test coverage for degraded node scenarios during upgrades and etcd recovery status conditions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

In response to this:

What this PR does / why we need it:

When the etcd recovery job fails but etcd self-heals, the EtcdRecoveryJobFailed condition was never cleared. This caused the OpenShift Console to display a stale error message ("Error in Etcd Recovery job: the Etcd cluster requires manual intervention.") on the HostedCluster overview page, even when the cluster was fully healthy (Available=True, Degraded=False, EtcdAvailable=True).

This fix adds two checks in reconcileETCDMemberRecovery:

• When a failed recovery job exists but the etcd StatefulSet is fully available (3/3 replicas), clean up the job and clear the condition.
• When no failing etcd pods exist and etcd is healthy, clear any stale EtcdRecoveryJobFailed condition.

Which issue(s) this PR fixes:

Fixes https://issues.redhat.com/browse/OCPBUGS-84577

Special notes for your reviewer:

• The etcd recovery feature is gated behind ENABLE_ETCD_RECOVERY env var and only applies to managed, highly-available etcd clusters.
• Both fix paths were verified on a live KubeVirt HCP cluster by simulating the stale condition and confirming it gets cleared.
• The etcd_manual_intervention_required metric (which reads this condition) will also correctly reset.

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Summary by CodeRabbit

• Bug Fixes

• In-place upgrade failures now report the degraded node name and its degraded-reason annotation.

• Etcd recovery no longer forces manual intervention when the cluster is healthy; recovery resources and failure condition are cleared when etcd is fully ready.

• Tests

• Added tests covering degraded-node upgrade errors and etcd recovery status transitions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Full review triggered.

Full review triggered.

Full review triggered.

In response to this:

/verified by me in local env

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

The root cause is a logic bug in setEtcdRecoveryCondition() at line 121 of etcd_recovery.go. The guard condition only triggers an update when the condition's Status field changes:

if oldCondition == nil || oldCondition.Status != condition.Status {
    meta.SetStatusCondition(&hcluster.Status.Conditions, condition)
    // ...
}

The PR adds logic to clear a stale EtcdRecoveryActive condition by setting it to Status=ConditionFalse with Reason=AsExpected. However, the stale condition already has Status=ConditionFalse (with Reason=EtcdRecoveryJobFailed). Since the Status field is identical (ConditionFalse → ConditionFalse), the guard clause short-circuits and the Reason is never updated.

This affects all three failing test subtests:

1. When_etcd_is_healthy_and_stale_EtcdRecoveryJobFailed_condition_exists — enters detectAndTriggerEtcdRecovery, finds no failing pods, etcd is fully available, detects stale condition, calls setEtcdRecoveryCondition(ConditionFalse, AsExpected) — skipped because Status unchanged.
2. When_failed_job_exists_but_etcd_recovered — enters handleExistingEtcdRecoveryJob, job failed but etcd has 3/3 replicas, calls setEtcdRecoveryCondition(ConditionFalse, AsExpected) — skipped because Status unchanged.
3. When_etcd_pods_have_restarted_but_recovered — same path as Merge from openshift-hive/openshift #1 through detectAndTriggerEtcdRecovery.

The codecov/project failure is a cascading consequence: the TestReconcileETCDMemberRecovery failures cause the entire hypershift-operator test shard to exit non-zero → the "Upload to Codecov" step is skipped → Codecov sees 85 fewer files → reports a -3.68% coverage drop → marks the project check as failed.

1. Fix the setEtcdRecoveryCondition guard clause to also trigger an update when the Reason changes, not just the Status. For example:

   if oldCondition == nil || oldCondition.Status != condition.Status || oldCondition.Reason != condition.Reason {
       meta.SetStatusCondition(&hcluster.Status.Conditions, condition)
       if err := r.Client.Status().Update(ctx, hcluster); err != nil {
           return fmt.Errorf("failed to update etcd recovery job condition: %w", err)
       }
   }

   Alternatively, compare the Message as well if message changes should also trigger updates, or simplify by always calling meta.SetStatusCondition (which already performs idempotency checks internally and only updates LastTransitionTime when Status changes).

2. Consider removing the guard entirely and relying on meta.SetStatusCondition's built-in behavior, which handles LastTransitionTime updates correctly. The Status().Update() call is cheap when nothing changes on the server side, and removing the guard eliminates this class of bugs.

3. Re-trigger the CI after the fix — the codecov/project failure will self-resolve once the unit tests pass and coverage data is uploaded.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.